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Abstract 

We analyze dynamical properties of the Shannon information entropy of a continuous probabil- 
ity distribution, which is driven by a standard diffusion process. This entropy choice is confronted 
with another option, employing the conditional Kullback-Leibler entropy. Both entropies dis- 
criminate among various probability distributions, either statically or in the time domain. An 
asymptotic approach towards equilibrium is typically monotonic in terms of the Kullback entropy. 
The Shannon entropy time rate needs not to be positive and is a sensitive indicator of the power 
transfer processes (removal/supply) due to an active environment. In the case of Smoluchowski 
diffusions, the Kullback entropy time rate coincides with the Shannon entropy "production" rate. 

PACS numbers: 02.50.-r, 05.40-a, 89.70,+c 

We consider diffusion processes in R with a constant or time-dependent diffusion coefficient D, 
characterized (non-uniquely) by space-time inhomogeneous probability densities p = p(x, t) and local 
velocity fields (drifts or currents). We specify the diffusive dynamics to refer to of a mass m particle 
in the external field of force, here taken to be conservative: F — F{x) = — VV. The associated 
Smoluchowski diffusion process with a forward drift b(x) = is analyzed in terms of the Fokker- 
Planck equation for the probability density p(x,t), 

d tP = DAp- V{b-p) (1) 

with the initial data po{x) = p(x, 0). We assume a priori the Einstein fluctuation-dissipation relation- 
ship D = ^7f, where (3 is interpreted as the friction (damping) parameter, T is the temperature of 
the bath, kg being the Boltzmann constant . 

With a solution p{x 1 i) of the Fokker-Planck equation, we associate the Shannon information 
entropy of a continuous probability distribution, also named differential entropy, 0J[S1I2] : 



S(t) = - J p(x,t) In p(x,t)dx (2) 
which is typically not a conserved quantity. The rate of change in time of S(t) readily follows. 
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Anticipating further discussion, let us stress that even in case of plainly irreversible diffusive 
dynamics, it is by no means obvious that the differential entropy should grow, decay (diminish) or 
show up a mixed behavior. Normally one tries, in terms of a suitable entropy definition, to justify the 
popular albeit tacit assumption that " generically" the time rate of "entropy" (whatever it is, be that 
S > 0) is non-negative. 

We take for granted the validity of standard assumptions which allow to interchange derivatives 
with indefinite integrals. Boundary restrictions upon p, vp and bp to vanish at spatial infinities (or at 
finite spatial volume boundaries) yield the rate equation: 



^=j[p(V. b ) + D.^f ]dx (3) 



which in case of b = refers to the one-dimensional free Brownian motion (Wiener process), when 
S > 0. The trivial example is provided by the familiar heat kernel p(x, t) = (47rDi) -1 / 2 exp(— x 2 /4Dt) 
whose differential entropy reads S(t) = (1/2) ln(47reDt), implying S — l/2i > 0. 

We can rewrite Eq. in a number of equivalent ways. To this end let us introduce the notion 
of the current velocity of Brownian particles in external force field (with a short-hand notation v = 
v(x,t)): v = b — u = -j^x — D^-£ which allows us to transform the Fokker-Planck equation into 
the continuity equation d t p — — V(i> • p) and back. Since (u 2 ) = —D(V ■ u), we have: DS = 
D(V • b) + (u 2 ) = D(V . v )=-{v- u). 

By reasons to become clear in below, we shall pay particular attention to the entropy balance 
equation in the form: 

DS = (v 2 ) -(b-v) (4) 

where (•) denotes the mean value with respect to p. 

This balance equation is extremely persuasive when recast in the thermodynamically motivated 
form: 

dt~\dt) in df U 
The non-negative term (S)i n = (l/D)(v 2 ) in Eq. Q may be interpreted as the measure of time rate 
at which the Shannon information entropy of the process shows a tendency to grow (is generated due 
to an implicit thermal agitation), at the expense of the environment. This excess "entropy production" 
by the environment may possibly be counterbalanced by the entropy/heat removal due to dissipation, 
provided we have Q > 0. The characteristic "power release" expression: 

in the formal thermodynamical lore is expected to refer to the time rate at which the mechanical 
work per unit of mass is dissipated (returned back to the reservoir) in the form of heat. Notice that 
ksTQ — J F ■ j dx, where b — Fj (m/3), j = vp and T stands for the temperature of the bath. 
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However, in the diffusion context, Q has an unspecified sign. It may be positive, with an in- 
terpretation of the power absorption by the environment. Q may take negative values, when the 
power is drained out from the environment. In below we shall discuss those issues for the exemplary 
Ornstein-Uhlenbeck process. 

Of particular interest is the case of a constant information entropy 5 = which amounts to the 
existence of steady states, |S]. In the simplest case, when the diffusion current vanishes, we encounter 
the primitive realization of the state of equilibrium with an invariant density p. Then, b = u = DV In p 
and we readily arrive at the classic equilibrium identity for the Smoluchowski process: 

-(l/fc B T)W = Vlnp (7) 

which determines the functional form of the invariant density in case of a given conservative force 
field, ^ [7] . There is an ample discussion in Ref . [5] of how these properties match with time reversal 
of the stationary diffusion process and the vanishing of the entropy production rate {S)i n . 

Coming back to the general discussion, let us define the so-called (fictitious) thermodynamic 
force Fth = v/D associated with the Smoluchowski diffusion and introduce its corresponding time- 
dependent potential function ^>{x,t): 

k B TF th =F-k B TV\np = -V^. (8) 

Notice that v = — (l/m/3)V v I'. In the absence of external forces (free Brownian motion), we 
obviously get F th = — Vlnp = — (1/D)u. 
The mean value of the potential 

* = V + k B T\np (9) 

associates with the diffusion process an obvious analogue of the thermodynamical Helmholtz free 
energy: 

(*) = (V)-TS G (10) 

where the dimensional (Gibbs-type) version S G = k B S of the information entropy has been introduced. 
The expectation value of the mechanical force potential (V) plays here the role of (mean) internal 
energy. 

By assuming that pVv vanishes at integration volume boundaries (or infinity), we easily get the 
time rate of Helmholtz free energy at a constant temperature T: 

j t (V) = -k B TQ-TS G . (11) 

By employing Eq. <|SJ), we readily arrive at 

| W — (»,r)(f)— (mfflW (12) 
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which either identically vanishes (equilibrium) or remains negative. 

Thus, Hclmholtz free energy either remains constant in time or decreases as a function of time at 



down to a finite minimum as t — > oo. 

However, this feature is a little bit deceiving. One should be aware that a finite minimum as well 
may not exist, which is the case e.g. for the free Brownian motion. Also, non-unique minima need to 
be excluded as well. 

Till now, we have deliberately avoided any use of the relative Kullback-Leibler entropy, pi EH El > 
which is often invoked for a comparison purpose to tell "how far from each other" two probability 
densities are. However, a reliability of the Kullback entropy may be questioned: this entropy fails to 
quantify properly a "comparison" of once given density function to itself, if considered at different 
instants of its non-stationary time evolution. 

To analyze defective features of the Kullback entropy in a fully controllable way, let us consider a 
one parameter family of Gaussian densities p a — p(x — a), with the mean a G R and the standard 
deviation fixed at a. They share the very same value iS CT = ^ \n(2irea 2 ) of the Shannon information 
(differential) entropy, independent of a. 

If we admit a to be an additional free parameter, a two-parameter family of Gaussian densities 

Pa ^ pa 

l<7 (x) appears. Such densities, corresponding to different values of a and a' do admit an 
" absolute comparison" in terms of the Shannon entropy: 



and the outcome is insensitive to translation parameters a and a' . 

By denoting a = a(t) and a' = <j{t') we can make any density amenable to the "absolute com- 
parison" formula at different time instants t' > t > 0. For the heat kernel we have a(t) — \j2Dt and 
therefore [a 1 jo) = y/¥Jt. 




There are many inequivalent ways to evaluate the "divergence" or "convergence" between prob- 
ability distributions. The relative (Kullback) entropy is typically used to quantify such divergence 
relative to the prescribed reference density, 0. 

We define the Kullback entropy IC(8,d') for a one-parameter family of probability densities pe, so 
that the " distance" between any two densities in this family can be directly evaluated. Let pgi denote 
the reference probability density. We have, : 



which, in view of the concavity of the function f(w) = -w In w, is positive. 

Let us indicate that the negative of IC, TL C = — /C, named the conditional entropy, is predominantly 
used in the literature [SHH|7] because of its affinity (regarded as a formal generalization) to the 



the rate set by the information entropy " production" Si„ . One may expect that actualy (^) (t) drops 




(13) 





(14) 
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differential entropy. Then e.g. one investigates an approach of —K towards its maximum (usually 
achieved at the value zero) when a running density is bound to have a unique stationary asymptotic, 

If we take 9' = 9 + A9 with A9 <C 1 , the following approximate formula holds true under a number 
of standard assumptions: 

IC{e,0 + A0)~ l -T e -(A9) 2 (15) 

where Tg denotes so-called Fisher information measure. With this proviso, we can evaluate the 
Kullback distance within a two-parameter (a, a) family of Gaussian densities, by taking 9 — > a. 
Passing to a 1 = a + Aa at a fixed value of a we arrive at: 

K( a ,a + Aa)~M. (16) 

For the record, we note that the respective Shannon entropies do coincide: S a — <S Q +Aa- 
Analogously, we can proceed with respect to the label a at a fixed: 

1C(a,a + Aa)~^¥- (17) 

when, irrespective of a: 

Sa+Aa — S a ~ . (18) 

a 



By choosing 9 — ► a 2 at a fixed, we get (now the variance a 2 is modified by its increment A(cr 2 )) 

4<7 4 



IC(a 2 ,a 2 + A(a 2 ))^ [ -^l (19) 



while 



which, upon identifications <r 2 = 2Z?i and A(<r 2 ) = 2D At, sets an obvious connection with the 
differential (A<S)(t) and thence with the time derivative S = l/2t of the heat kernel differential 
entropy. 

Our previous observations are a special case of more general reasoning. Namely, if we consider a 
two-parameter 9 = (0 i, #2) family of densities, then instead of Eq. (|15fl we would have arrived at 

JC(0, 9 + A9) ~ i 2 ^ • A9iA9j (21) 

where z, j, = 1, 2 and the Fisher information matrix is given as follows 

/" dliipg dlnpg 

• ; ' J"-— (22) 

In case of Gaussian densities, labelled by independent parameters 9\ = a and 82 = (alternatively 
02 = c 2 ), the Fisher matrix is diagonal and defined in terms of previous entries T a and T a (or T a i). 
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It is useful to note (c.f. also that in self-explanatory notation, for two 9 and 9' Gaussian 
densities there holds: 

m 9') = In - + - 1) + i(« - «') 2 (23) 

The first entry in Eq. (|23[1 coincides with the "absolute comparison formula" for Shannon entropies, 
Eq. . However for |#' — #| <C 1, the second term dominates the first one. 

Indeed, let us set oi = a and consider a 2 = 2Dt, A(cr 2 ) = 2D At. Then S(ct') - S(a) ~ At/2t, 
while K{9,9') ~ (At) 2 /4t 2 . Although, for finite increments At we have 

At 



(24) 

the time derivative can be defined exclusively for the differential entropy, iS, and is meaningless in 
terms of the Kullback " distance" . 

Let us mention that no such obstacles arise in the standard cautious use of the relative Kullback 
entropy TL C . Indeed, normally one of the involved densities stands for the stationary reference one 
pe'{x) = p*(x), while another evolves in time pe(x) = p(x, t), t £ R + , thence we need: 

H c (t) = -K(p t \p^ . (25) 

In the presence of external forces, the property Eq. 1)12(1 may consistently quantify an asymptotic 
approach towards a minimum corresponding to an invariant (presumed to be unique) probability 
density of the process. Indeed, by invoking Eq. J7J) we realize that 

P 4 x ) = Lex V (-p4) (26) 



where Z — J exp(-V(x)/k B T)dx, sets the minimum of (*)(t) at = = -k B T\nZ. 

Let us take the above p*{x) as a reference density with respect to which the divergence of p(x,t) 
is evaluated in the course of the pertinent Smoluchowski process. This divergence is well quantified 
by the conditional Kullback entropy Tt c (t), where: 

H c (t) = - J p In (j-J dx = S(t)-\nZ-^. (27) 

Consequently, in view of Eqs. and JSJl, we get 

He = S + Q = (S) m > (28) 

so that 4i{^) = —{ksT) i-L c - An approach of (^)(t) towards the minimum proceeds in the very same 
rate as this of Ti. c (t) towards its maximum. 

In contrast to Ti. c which is non-negative, we have no growth guarantee for the differential en- 
tropy, since the sign of S is unspecified. Nonetheless, the balance between the time rate of entropy 
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production/removal and the power release into or out of the environment, is definitely correct and 
informative. 

We have S > — Q and surely <2 < ^ S > 0. If Q > 0, S may take negative values down to the 
lower bound — Q. 

It is quite illuminating to exemplify previous considerations by a detailed presentation of the 
standard one-dimensional Ornstein-Uhlenbeck process. We denote b(x) = —jx with 7 > and choose 
an initial density in the Gaussian form, with the mean value ao and variance a 2 . The Fokker-Planck 
evolution Eq. Q preserves the Gaussian form of p(x, t) while modifying the mean value and variance 
according to a(t) — ao exp(— 7 t), and 

a 2 (t) = (Jq exp(-2 7 i) + — [1 - expf-2-yt)] . (29) 
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Accordingly, since a unique invariant density has the form p* = y // j/2TrD exp(—jx 2 /2D) we obtain, 

m 

2 

H c (t) = cxp(-2 7 <)H c (po, P*) = exp(-2 7 i) (30) 

while in view of our previous considerations, we have S(t) = (1/2) ln[27recr 2 (i)] and T = l/a 2 (t). 
Therefore 

2 7 p - 7 qg) exp(-2 7 t) 
^-(^- 7 a 2 )exp(-2 7 t) 

We observe that ctq > D/j —> S < 0, while ctq < D/j — > S > 0. In both cases the behavior of 
the differential entropy is monotonic, although its ultimate growth or decay do critically rely on the 
choice of Ctq. Irrespective of cr 2 . the asymptotic value of S(t) as t — > 00 reads (1/2) ln[27re(D/ 7 ). 

The differential entropy evolution is anti-correlated with the Fisher measure of the probability 
localization, since 

jS_ 

[£-(,D- 7 a 2 )exp(-2 7 t)P 
For all <t 2 the asymptotic value of T reads j/D. 

We have here a direct control of the behavior of the "power release" expression Q = Ti c — S. Since 

U c = (l 2 a 2 /D) exp(-2 7 i) > , (33) 

in case of S < we encounter a continual power removal Q > into the thermal environment. 

In case of S > the situation is more complicated. For example, if ao = 0, we can easily check 
that Q < 0, i.e. we have the power supply from the environment for all t G R + . More generally, the 
sign of Q is negative for a 2 , < 2(D — 7 c§)/ 7 . If the latter inequality is reversed, the sign of Q is not 
uniquely specified and suffers a change at a suitable time instant tchangeio-o, c 2 ). 

Standard notions of thermodynamical entropy are basically not considered in the time domain. 
However any conceivable idea of "approaching" the state of equilibrium or passing from one such 
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state to another (steady) state, surely involves the time dependence and the related non-equilibrium 
dynamical process. This refers to attempts to give a precise meaning to Boltzmann's H-theorem 
under non-equilibrium conditions and search for an origin of increasing entropy in terms of model 
systems. Consult e.g. at this point, both standard motivations and apparent problems encountered 
in connection with the H-theorem and its diffusion process analogues, 0IE10- 

Our analysis of simple diffusion-type models indicates that the very notion of entropy is non- 
universal and purpose-dependent. In particular, although the conditional Kullback entropy is often 
considered as as the only valid "entropy growth" justification, a deeper insight into the underlying 
physical phenomena (power transfer processes) is available only through the differential (Shannon) 
entropy notion. 
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